The present disclosure relates in general to the use of computer systems to model biological neural systems. More specifically, the present disclosure relates to modeling biological neural systems using computer systems that implement a pulsed neural network having temporal encoders and informative looping signals.
An animal's continually evolving perceptions of and behavioral responses to dynamic sensory stimuli must ultimately be derived from information contained within relatively brief segments of so-called spike trains generated by neurons. These spike trains, also known as pulses or action potentials, are voltage spikes that can travel down a nerve fiber. A typical neuron-generated spike train includes spatiotemporal patterns comprising localized events with a distinctive spatial and temporal structure. Although these events occur on a wide variety of spatial and temporal scales, a biological neural system such as the brain can integrate them and extract relevant pieces of information. Thus, biological neural systems impose order on information provided by noisy spatiotemporal sensory inputs.
The neuron is a central element of biological neural systems. The bulk of the brain is made up of structural cells known as glial cells and astrocytes. Lying among these structural cells are neurons that conduct electrical impulses along fibers. It has been estimated that the average human brain contains about 100 billion neurons and, on average, each neuron is connected to about 1000 other neurons. These neuron and their interconnections form vast and complex biological neural networks that are the mainstay of the brain's processing capabilities. Neurons are remarkable among the cells of the body in their ability to propagate signals such as spike trains or action potentials rapidly over large distances. So-called sensory neurons change their activities by firing sequences of action potentials in various temporal patterns in response to the presence of external stimuli, such as light, sound, taste, smell and touch. Information about a stimulus is encoded in this pattern of action potentials and transmitted into and around the brain.
In biological neural systems, the point of contact between an axon of one neuron and a dendrite of another neuron is called a synapse, and, with respect to the synapse, the two neurons are respectively called pre-synaptic and post-synaptic. Neurons, when activated by sufficient inputs received via synapses, emit spikes that are delivered to those synapses to which the neuron is pre-synaptic. Neurons can be either “excitatory” or “inhibitory.” Synaptic conductance is a measure of the amount of influence a synapse will have on its post-synaptic target when the synapse is activated by a pre-synaptic spike. The synaptic conductance can change with time as a function of the relative spike times of pre-synaptic and post-synaptic neurons, as per spike-timing dependent plasticity (STDP). The standard STDP rule increases the conductance efficacy of a synapse if its post-synaptic neuron fires after its pre-synaptic neuron fires, and decreases the conductance efficacy of a synapse if the order of the two firings is reversed. A person's mental possession of individual experiences is stored in the conductance of signals across the trillions of synapses throughout the brain.
The computations underlying all aspects of the operation of a biological neural system are carried out within the context of the neural code with which the relevant information is represented in the spike trains. A determination of the information coding schemes used within biological neural systems is useful, not only because of the intrinsic interest in the nature of the neural code itself but also because of the constraints that knowledge of the code can place on the development of models for the mechanisms underlying biological neural computations.
Evolution has provided the human brain with many desirable characteristics not present in contemporary computer systems, including but not limited to massive parallelism, distributed representation and computation, learning ability, generalization ability, adaptability, inherent contextual information processing, fault tolerance and low energy consumption. Contemporary digital computers outperform humans in the domain of numeric computation and related symbol manipulation. However, humans can effortlessly solve complex perceptual problems (e.g., recognizing an acquaintance in a crowd from a mere glimpse of the person's face) at a speed and to an extent that dwarfs the fastest computer. A reason for such a remarkable difference in the performance of biological neural systems and computers is that the biological neural system architecture is completely different from a typical computer system architecture. This difference significantly affects the types of functions each computational model can best perform.
Artificial neural networks (ANNs), also known as neuromorphic and synaptronic systems, are computational systems that permit electronic systems to essentially function in a manner analogous to that of biological neural systems such as a human brain. In the present disclosure, any reference to an ANN is used in the broadest sense to cover a wide range of computer and electronic systems that model biological neural networks, including but not limited to pulsed neural networks (PNNs). ANNs do not generally utilize the traditional digital model of manipulating zeros and ones. Instead, a typical ANN creates connections between processing elements that are roughly functionally equivalent to neurons of a biological brain.
ANNs are often implemented as relatively large, parallel computing systems having many simple processors with many interconnections. Modeling a biological nervous system using ANNs (or PNNs) can contribute to a better understanding of biological functions. Many ANN models attempt to use some “organizational” principles believed to be used in the human brain. Contemporary computer hardware technology (such as VLSI and optical) has made such modeling feasible. ANNs incorporate knowledge from a variety of disciplines, including neurophysiology, cognitive science/psychology, physics (statistical mechanics), control theory, computer science, artificial intelligence, statistics/mathematics, pattern recognition, computer vision, parallel processing and hardware (e.g., digital/analog/VLSI/optical).
A contemporary ANN is comprised of a network of artificial neurons (also known as “nodes”). These nodes are connected to each other, and the strength of their connections to one another is assigned a value. For example, connections may be identified as exhibiting inhibition (maximum inhibition being negative 1.0) or excitation (maximum excitation being plus 1.0). If the magnitude value of the connection is high, this indicates a strong connection. Within each node's design, a transfer function is also built in. There are three types of neurons in a typical ANN, namely input nodes, hidden nodes and output nodes.
Input nodes take in information that can be numerically expressed. The information is presented as activation values, wherein each node is given a number, and a number having a higher magnitude causes a greater activation. This information is then passed throughout the network. Based on the connection strengths or efficacy (weights), inhibition or excitation, and transfer functions, the activation value is passed from node to node. Each node sums the activation values it receives. Each node than modifies the value based on its transfer function. The activation flows through the network, through hidden layers, until it reaches the output nodes. The output nodes then reflect the input in a meaningful way to the outside world.
There are many types of neural networks, but the two broadest categories are feed-forward and feed-back networks. A feed-forward network is a non-recurrent network having inputs, outputs, and hidden layers. The signals can only travel in one direction. Input data is passed onto a layer of processing elements that perform calculations. Each processing element makes its computation based upon a weighted sum of its inputs. The new calculated values then become the new input values that feed the next layer. This process continues until it has gone through all the layers and determined the output. A threshold transfer function is sometimes used to quantify the output of a neuron in the output layer.
A feed-back network includes feed-back paths, which means that their signals can travel in both directions using loops. All possible connections between neurons are allowed. Because loops are present in this type of network, under certain operations, it may become a non-linear dynamical system that changes continuously until it reaches a state of equilibrium. Feed-back networks are often used in associative memories and optimization problems, wherein the network looks for the best arrangement of interconnected factors.
In an ANN, a spike generation function may be modeled by a component known generally as a temporal encoder or a spike generator. Thus, the ability to gain greater knowledge of the previously describe biological neural system depends on the development of ANNs that model how neural information is encoded in recurrent networks of spiking temporal encoders. Temporal encoders provide an alternative to both digital and analogue encoding of information by integrating received signals asynchronously and producing subsequent sets of asynchronous pulsed signals. Certain configurations of temporal encoder networks allow for extraction of information about sets of signals input to the network from sets of pulses or spikes output from the network.